Skip to content

Fix manifest validation issue #4541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
IanButterworth wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Fix manifest validation issue #4541

IanButterworth wants to merge 4 commits into from
+8 −0

Conversation

@IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Nov 26, 2025 •
edited
Loading

Fixes #4540

The issue isn't obvious to me, but this is the best Claude Opus 4.5 could come up with

Bug Analysis: Missing Manifest Entry Error

Error Message

`LLVM_jll=...` depends on `libLLVM_jll=...`, but no such entry exists in the manifest.

Also seen with: Enzyme → ADTypes

Key Observations

  • Intermittent - doesn't always happen
  • Fresh depots affected - not just stale cache
  • Affects: Julia 1.12.x, 1.13.0-alpha1, nightly
  • Cleaning depot fixes it (for cached cases)

Root Cause

When building deps_map for manifest entries, query_deps_for_version returns ALL deps including weak deps:

deps_for_version = Registry.query_deps_for_version(
    deps_map_compressed, weak_deps_map_compressed,  # ← includes weak deps!
    pkg.uuid, pkg.version
)

But weak deps that weren't triggered (no extension trigger present) are NOT in vers/pkgs because the resolver marked them as "uninstalled".

Result: entry.deps contains UUIDs that have no corresponding manifest entry.

Why Existing Fix Doesn't Help

Commit cdc17a0d7 ("allow having unknown weak dependencies") handles weak deps that are not in any registry. It filters them from fixed before resolution.

This bug is about weak deps that ARE registered but weren't resolved because the trigger wasn't needed.

The Fix

Filter deps_map to only include UUIDs that are actually in pkgs:

pkgs_uuids = Set{UUID}(pkg.uuid for pkg in pkgs)

for uuid in deps_for_version
    uuid in pkgs_uuids || continue  # ← Skip deps not in manifest
    d[names[uuid]] = uuid
end

Why Intermittent?

Unclear. Possibly related to:

  • Registry caching/state
  • Which package versions get selected
  • Timing of when weak deps get evaluated

@giordano

@github-project-automation github-project-automation bot moved this to New in Pkg.jl Nov 26, 2025
@giordano
Copy link
Member

giordano commented Nov 26, 2025

Sadly this isn't very easy to test because we don't have a simple reproducer, I only saw this on CI, not locally. One could try to jump on those machines with tmate, but I haven't had a chance to do that (CC @vchuravy in case you have spare time for this)

@IanButterworth IanButterworth mentioned this pull request Dec 1, 2025
Open
@vchuravy
Copy link
Member

vchuravy commented Dec 9, 2025

@IanButterworth

JULIA_DEPOT_PATH=$(mktemp -d) julia +1.13 --project=. -e 'import Pkg; Pkg.test(;coverage=true, julia_args=["--check-bounds=yes", "--compiled-modules=yes", "--depwarn=yes"], test_args=["--verbose"], force_latest_compatible_version=false, allow_reresolve=true)'

reproduces for me:

 Installing known registries into `/tmp/tmp.svyhbxsq7L`
       Added `General` registry to /tmp/tmp.svyhbxsq7L/registries
   Installed LibTracyClient_jll ─ v0.9.1+6
   Installed LLVMExtra_jll ────── v0.0.38+0
   Installed CEnum ────────────── v0.5.0
   Installed ExprTools ────────── v0.1.10
   Installed PrecompileTools ──── v1.3.3
   Installed GPUCompiler ──────── v1.7.5
   Installed LLVM ─────────────── v9.4.4
   Installed StructIO ─────────── v0.3.1
   Installed Enzyme_jll ───────── v0.0.226+0
   Installed Reexport ─────────── v1.2.2
   Installed Tracy ────────────── v0.1.6
   Installed ObjectFile ───────── v0.5.0
   Installed JLLWrappers ──────── v1.7.1
   Installed Scratch ──────────── v1.3.0
   Installed Preferences ──────── v1.5.0
  Installing artifacts ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3/3
    Updating `~/src/Enzyme/Project.toml`
  [fa961155] + CEnum v0.5.0
  [f151be2c] + EnzymeCore v0.8.17 `lib/EnzymeCore`
  [61eb1bfa] + GPUCompiler v1.7.5
  [929cbde3] + LLVM v9.4.4
  [d8793406] + ObjectFile v0.5.0
  [aea7be01] + PrecompileTools v1.3.3
  [21216c6a] + Preferences v1.5.0
⌅ [7cc45869] + Enzyme_jll v0.0.226+0
  [b77e0a4c] ~ InteractiveUtils ⇒ v1.11.0
  [8f399da3] ~ Libdl ⇒ v1.11.0
  [37e2e46d] ~ LinearAlgebra ⇒ v1.13.0
  [de0858da] ~ Printf ⇒ v1.11.0
  [9a3f8284] ~ Random ⇒ v1.11.0
  [2f01184e] ~ SparseArrays ⇒ v1.13.0
    Updating `~/src/Enzyme/Manifest.toml`
  [fa961155] + CEnum v0.5.0
  [f151be2c] + EnzymeCore v0.8.17 `lib/EnzymeCore`
  [e2ba6199] + ExprTools v0.1.10
  [61eb1bfa] + GPUCompiler v1.7.5
  [692b3bcd] + JLLWrappers v1.7.1
  [929cbde3] + LLVM v9.4.4
  [d8793406] + ObjectFile v0.5.0
  [aea7be01] + PrecompileTools v1.3.3
  [21216c6a] + Preferences v1.5.0
  [189a3867] + Reexport v1.2.2
  [6c6a2e73] + Scratch v1.3.0
  [53d494c1] + StructIO v0.3.1
  [e689c965] + Tracy v0.1.6
⌅ [7cc45869] + Enzyme_jll v0.0.226+0
  [dad2f222] + LLVMExtra_jll v0.0.38+0
  [ad6e5548] + LibTracyClient_jll v0.9.1+6
  [0dad84c5] + ArgTools v1.1.2
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [f43a241f] + Downloads v1.7.0
  [7b1f6079] + FileWatching v1.11.0
  [b77e0a4c] ~ InteractiveUtils ⇒ v1.11.0
  [ac6e5ff7] + JuliaSyntaxHighlighting v1.12.0
  [4af54fe1] + LazyArtifacts v1.11.0
  [b27032c2] + LibCURL v1.0.0
  [76f85450] + LibGit2 v1.11.0
  [8f399da3] ~ Libdl ⇒ v1.11.0
  [37e2e46d] ~ LinearAlgebra ⇒ v1.13.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [ca575930] + NetworkOptions v1.3.0
  [44cfe95a] + Pkg v1.13.0
  [de0858da] ~ Printf ⇒ v1.11.0
  [9a3f8284] ~ Random ⇒ v1.11.0
  [ea8e919c] + SHA v1.0.0
  [9e88b42a] + Serialization v1.11.0
  [2f01184e] ~ SparseArrays ⇒ v1.13.0
  [f489334b] + StyledStrings v1.11.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [e66e0078] + CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] + LibCURL_jll v8.16.0+0
  [e37daf67] + LibGit2_jll v1.9.1+0
  [29816b5a] + LibSSH2_jll v1.11.3+1
  [14a3606d] + MozillaCACerts_jll v2025.11.4
  [4536629a] + OpenBLAS_jll v0.3.29+0
  [458c3c95] + OpenSSL_jll v3.5.4+0
  [efcefdf7] + PCRE2_jll v10.46.0+0
  [bea87d4a] + SuiteSparse_jll v7.10.1+0
  [83775a58] + Zlib_jll v1.3.1+2
  [3161d3a3] + Zstd_jll v1.5.7+1
  [8e850b90] + libblastrampoline_jll v5.15.0+0
  [8e850ede] + nghttp2_jll v1.67.1+0
  [3f19e933] + p7zip_jll v17.7.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
     Testing Enzyme
ERROR: `LLVM_jll=86de99a1-58d6-5da7-8064-bd56ce2e322c` depends on `libLLVM_jll=8f36deef-c2a5-5394-99ed-8e07531fb29a`, but no such entry exists in the manifest.
Stacktrace:
  [1] pkgerror(::String, ::Vararg{String})
    @ Pkg.Types ~/.julia/juliaup/julia-1.13.0-alpha2+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/Types.jl:68
  [2] validate_manifest(julia_version::VersionNumber, project_hash::Base.SHA1, manifest_format::VersionNumber, stage1::Dict{String, Vector{Pkg.Types.Stage1}}, other::Dict{String, Any}, registries::Dict{String, Pkg.Types.ManifestRegistryEntry})
    @ Pkg.Types ~/.julia/juliaup/julia-1.13.0-alpha2+0.x64.linux.gnu/share/julia/stdlib/v1.13/Pkg/src/manifest.jl:198

Important was to remove the Manifest.toml in my Enzyme checkout

@IanButterworth
Copy link
Member Author

IanButterworth commented Dec 9, 2025

Thanks. I assume you've checked out EnzymeAD/Enzyme.jl@a519c3f (#2815) there?

@vchuravy
Copy link
Member

vchuravy commented Dec 9, 2025

No need, I was on main for this, but that branch would also work.

@IanButterworth
Copy link
Member Author

IanButterworth commented Dec 9, 2025

I can reproduce it locally and this PR fixes it. All I did between testing is delete the manifest. Same depot.

@IanButterworth IanButterworth marked this pull request as ready for review December 9, 2025 15:38
@IanButterworth IanButterworth moved this from New to In review in Pkg.jl Dec 9, 2025
@IanButterworth IanButterworth changed the title Tryfix manifest validation issue Fix manifest validation issue Dec 9, 2025
@giordano
Copy link
Member

giordano commented Dec 9, 2025

Do you have an idea of could have suddenly caused this? Could have been a change in the registry?

@IanButterworth
Copy link
Member Author

IanButterworth commented Dec 9, 2025

Not me, but Claude. Does the following make sense?

Yes, this could absolutely be caused by registry state changes. Here's how:

Registry-dependent scenarios that could trigger this bug:

  1. A new version of a package adds a weak dependency: If LLVM_jll releases a new version that adds libLLVM_jll as a weak dep (for an extension), and the resolver selects that version, the bug manifests. The intermittency comes from which version gets resolved.

  2. An existing package version gets a weak dep registered: If someone registers a new version of a dependency package that becomes the extension trigger, and suddenly a weak dep becomes "visible" to query.

  3. Timing of registry updates: Fresh depots get the latest registry state. If a package recently added a weak dep, users with older cached registries might not see the issue yet.

Let me verify this by checking when LLVM_jll added libLLVM_jll as a weak dependency:

Using "Fetch Web Page"

Fetched https://raw.githubusercontent.com/JuliaRegistries/General/master/L/LLVM_jll/WeakDeps.toml

Using "Fetch Web Page"

Fetched https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/LLVM/LLVM_full%40v19/bundled/Project.toml

Using "Fetch Web Page"

Fetched https://github.com/JuliaPackaging/Yggdrasil/tree/master/L/LLVM

Based on my analysis, yes, this bug is almost certainly dependent on registry state. Here's the detailed explanation:

Why this is registry-dependent:

The Root Cause

The bug occurs when:

  1. A package (like LLVM_jll) has a weak dependency on another package (like libLLVM_jll)
  2. The weak dep is used for a package extension
  3. The extension's trigger package is NOT in the dependency graph
  4. But query_deps_for_version() returns ALL deps including weak deps

Registry State Triggers

  1. New version with weak dep: If a new version of LLVM_jll was registered that added libLLVM_jll as a weak dependency (for an extension), and the resolver selects that version, the bug manifests.

  2. Which version gets resolved matters: The bug is intermittent because:

    • Older versions of a package may not have the weak dep
    • Resolution might select different versions based on compat constraints
    • Fresh depots always get latest registry → more likely to resolve to newer versions with weak deps

  3. The LLVM_jll / libLLVM_jll case: These are JLL packages built by Yggdrasil. If a recent LLVM_jll version added libLLVM_jll as a weak dep (likely for an extension that loads it conditionally), that's the trigger.

@IanButterworth IanButterworth mentioned this pull request Dec 11, 2025
Open
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport 1.10 backport 1.12 backport 1.13

Projects

Pkg.jl
Status: In review

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

LLVM_jll=86de99a1-58d6-5da7-8064-bd56ce2e322c depends on libLLVM_jll=8f36deef-c2a5-5394-99ed-8e07531fb29a, but no such entry exists in the manifest.

3 participants

@IanButterworth @giordano @vchuravy